D-Score: Holistic Dialogue Evaluation Without Reference

In artistic gymnastics, difficulty score or D-score is used for judging performance. Starting from zero, an athlete earns points different aspects such as composition requirement, difficulty, and connection between moves. The final a of the quality various performance indicators. Similarly, when evaluating dialogue responses, human judges generally follow number criteria, among which language fluency, context coherence, logical consistency, semantic appropriateness are on top agenda. this paper, we propose automatic evaluation framework called that resembles way gymnastics evaluated. Following four criteria above, devise range tasks model them under multi-task learning framework. proposed framework, without relying any human-written reference, learns to appreciate overall human-human conversations through representation shared by all over-fitting individual task domain. We evaluate performing comprehensive correlation analyses with judgement three datasets, two past DSTC series, benchmark against state-of-the-art baselines. not only outperforms best baseline large margin in terms system-level Spearman but also represents important step towards explainable scoring.